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The theory of center-surround interaction in non-classical receptive field can be applied in night vision 
information processing. In this work, an optimized compound receptive field modulation method is pro¬ 
posed to extract salient contour from complex natural scene in low-light-level (LLL) and infrared images. 
The kernel idea is that multi-feature analysis can recognize the inhomogeneity in modulatory coverage 
more accurately and that center and surround with the grouping structure satisfying Gestalt rule 
deserves high connection-probability. Computationally, a multi-feature contrast weighted inhibition 
model is presented to suppress background and lower mutual inhibition among contour elements; a 
fuzzy connection facilitation model is proposed to achieve the enhancement of contour response, the 
connection of discontinuous contour and the further elimination of randomly distributed noise and tex¬ 
ture; a multi-scale iterative attention method is designed to accomplish dynamic modulation process and 
extract contours of targets in multi-size. This work provides a series of biologically motivated computa¬ 
tional visual models with high-performance for contour detection from cluttered scene in night vision 
images. 

© 2014 Elsevier B.V. All rights reserved. 


1. Introduction 

Contour extraction plays an important role in night vision 
images (low-light-level (LLL) and infrared images) understanding. 
Most applications of night vision target detection and recognition 
are in outdoor scene, which contains a great deal of natural texture 
like grass or foliage. The traditional edge detection operators, such 
as Canny, retain many non-contour boundaries. So how to elimi¬ 
nate uninteresting local boundaries engendered from textures 
and maintain the integrated contour are the major problems in 
night vision contour extraction. 

Many solutions to extract contour from complex scene are pro¬ 
posed. In those methods, contour extraction through non-classical 
receptive field model based on the mechanism of biological vision 
[1] is distinguished by remarkable performance in the high-quality 
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visible images. The area beyond classical receptive field (CRF) of neu¬ 
rons in primary visual cortex (VI), namely non-classical receptive 
field (nCRF), can modulate the stimuli response in CRF [2-10]. It 
has been found that the majority of the surround modulation is sup¬ 
pressive, which makes the isolated contour more significant than 
homogeneous background. Grigorescu et al. [2,3] proposed a series 
of related nCRF inhibition models to extract contour, including iso¬ 
tropic and anisotropic inhibition models [4], which utilize surround 
suppression in center direction to weaken natural texture. Giuseppe 
et al. [5] developed a multi-resolution contour detection method 
based on Bayesian denoising and a surround inhibition technique. 
The model built by Sang et al. [6] with butterfly-shaped subregion, 
weighting inhibition by the difference of stimulus-direction be¬ 
tween CRF and nCRF, can decrease the impact of collinear suppres¬ 
sion. Ursino et al. [7,8] introduced level attention mechanism to 
gain contour in different scales. Zeng et al. [9,10] improved the 
butterfly-shaped inhibition to present a two-scale based contour 
extraction model and an orientation-selective inhibition model. 
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These approaches work well on suppressing background in vis¬ 
ible images with high resolution, as is shown in Fig. 1, but they do 
not always perform well in LLL and infrared images. Compared 
with high-quality visible image, heavier noise and lower contrast 
in night vision images lead to inaccurate and cracked contour, 
and it is difficult to suppress natural texture. So in this paper, we 
design a reasonable inhibition model for contour extraction in 
night vision image, namely multi-feature contrast (MFC) weighted 
inhibition model. 

On the other hand, inhibition weakens the strength of contour 
response. Contours in night vision image are prone to disconnect 
due to imaging characteristics, surround inhibition and noise inter¬ 
ference, which will affect the subsequent target recognition. 

Some physiological findings 11] have shown that modulations 
from nCRF are both inhibition and facilitation. Tang et al. [12,13] 
added collinear excitation in nCRF inhibition model, which calcu¬ 
lates facilitation based on curvature in grouping properties to en¬ 
hance the response of contour points. In the aspect of contour 
grouping, Sha et al. [14] proposed a significance calculation meth¬ 
od based on curvature change and developed a grouping algorithm 
to extract remarkable curve. Geisler et al. [15] testified that con¬ 
tour grouping in natural scene corresponds with the co-circularity 
rule by studying the statistical properties of contours in natural 
image, and carried out a maximum likelihood local grouping func¬ 
tion for contour. Elder et al. [16] studied the statistical properties of 
contour perceptual organization in Gestalt rules, including proxim¬ 
ity, continuity and similarity, and suggested a Bayesian ratio model 
to detect contour. Wang et al. [17,18] provided a ratio-contour 
method to extract significant closed contour based on graph mod¬ 
el. Guy et al. 19-21] developed a global cognitive model, namely 
tensor voting, to infer significant structures in image by tensor 
analysis and neighbor voting on each point. These contour group¬ 
ing models devote to organize local segments to global contour 
information. 

However, it is difficult to realize accurate contour connection 
through single grouping rule. In this paper, more contour grouping 
constraints are introduced to consider facilitation mechanism. We 
propose a fuzzy connection (FC) facilitation model to enhance 
weak contours and connect discontinuous contours to keep the 
integrated salient contours. 

A multi-scale iterative attention method proposed in this paper, 
combining the MFC weighted inhibition model and FC facilitation 
model, realizes the optimized compound modulation. It primarily 
solves the problems of surrounding suppression and contour con¬ 
nection in LLL and infrared images. 


The rest of this paper is organized as follows. Section 2 details 
our method of salient contour extraction in night vision images. 
Section 3 discusses the analysis of presented models. Section 4 
gives the practical application and performance of our method in 
LLL and infrared images in nature scenes. Finally, Section 5 
summarizes the main conclusions of this paper and the future 
works. 

2. Visual model of salient contour extraction in night vision 

2.1. nCRF model 

Two-dimensional Gabor function is used to describe the recep¬ 
tive field profile of simple cells in visual cortex. The response mod¬ 
ulus of odd-even pair of Gabor filters, called Gabor energy, can 
effectively models fundamental characteristics of typical complex 
cells, which can be seen as local orientation energy operators 
[22]. The maximum response of complex cell is used to accurate 
positioning edge and line in graphics. So Gabor energy is chosen 
to simulate the activities of complex cell. Two-dimensional Gabor 
filter can be expressed as follow. 

Sxacp(x,r,9) = exp (- * + 2 y ) • cos ( 27C |+ v) 0) 

where x = x cos 9 + y sin 9, y = -x sin 9 + y cos 9 , in which 6 is the 
preferred orientation of CRF; cp is phase offset; g Aa(p (x,y;0) with 
(p = 0 and cp = 7i/2 represent the even and odd Gabor filter 
respectively; y is the spatial aspect ratio, which determines the 
eccentricity of the Gaussian envelope; X is wavelength; a is the 
standard deviation of Gaussian factor, which decides the area of 
the Gabor simulated CRF; o/X represents the spatial frequency 
bandwidth. 

Response of simple cells r kaB(p is given as the convolution of Ga¬ 
bor function and image /. Neuronal response to stimulus in CRF 
E a (x,y ; Oi) is defined as the combination of orthogonal pair of sim¬ 
ple cells’ responses. 

tW*,y; 9) = I(x,y) *g Xav (x,y-, 9) (2) 

4(x,y; 9,) = rj a0 (x,y e t ) + f\ an/1 {x,y, 9 t ) (3) 

where Gabor energy of CRF is divided into N e directions: 
Qi = in/N 9 , i = 0,1,... ,N e - 1. According to [9] we set N e = 12, 
y = 0.5, o/X = 0.56. o is a variable in our experiments. Output re¬ 
sponse of CRF is the maximum of E a {x,y\ Oi) over all orientations. 



Fig. 1 . Results of contour extraction in natural images. The columns from lift to right represent the original images, outputs of canny, butterfly-shaped inhibition [6], 
orientation-selective inhibition [10] and two-scale inhibition [9] models, respectively. 
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E„(x,y) = max{£ <r (x,y;0 j )|i = 0,1, ...,JV 0 - 1} (4) 

Physiological studies show that visual cortex cells are inhibi¬ 
tory principally. The surround inhibition is used to segment 
inhomogeneous region and highlight boundary. In those previous 
vision-based models [5,6,10], a special distance weighting func¬ 
tion co G {x,y) is usually used to describe the spatial structure of 
surround inhibition. The weighting function cu a (x,y) is defined 
as a half-wave rectified and LI-normalized difference of two 
concentric Gaussian functions D0G a j <(7 (x, y). 


DOGfjka (x,y) 


1 / x 2 +y 2 \ 

v / 2^Gf exp v w&f) 


"vlb exp 




(5) 


co ff (x,y) = |DOC <r , t<7 (x,y)| + /ll|DOG (7 , te (x,y)| + [| 1 , |z| + ={°^° (6) 

The inhibition term t a (x,y, Oi) is defined as the distance 
weighted local orientation energy E a (x,y; Oi) in annular surround 
of each pixel. 

ta (x,y; 9i) = co ff (x,y) * E ff (x,y; Si) (7) 

Here a and ko determine the area of CRF and nCRF respectively. 
We set k = 3 in our experiments, because that nCRF diameter may 
reach 2-5 times than CRF diameter. |DOG| + ensures that cu a (x,y) 
only works on nCRF region as is shown in Fig. 2. Radius r c of 
CRF area S c is got by setting DOG aik(T (x,y) = 0. Thus 

r c = \J 2 \nk/(k 2 - 1) • ka and the radius of nCRF area S nc is kr c . 


2.2. New method for salient contour extraction in night vision images 


The new method in this paper is shown in Fig. 3. The presented 
multi-feature contrast weighted inhibition model and fuzzy con¬ 
nection facilitation model is executed synchronously to suppress 
background texture and connect of discontinuous contour. Input 
the results of inhibition and facilitation into the multi-scale itera¬ 
tive attention method, we accomplish dynamic modulation and ex¬ 
tract contours of targets in multi-size. It performs well in salient 
contour extraction from night vision images in clutter nature 
scenes. 


2.3. MFC inhibition model 

The isotropic annular suppression structure mentioned above 
may cause mutual inhibition among inhomogeneous regions, 
which will result in weak contour. Butterfly-shaped and orienta¬ 
tion-selective models [6,9,10] weight inhibition by orientation dif¬ 
ference of CRF and nCRF to reduce the response in homogeneous 
region and maintain the response in inhomogeneous region. But 
they are susceptible to noise and easy to get inaccurate orientation 
in center and surround. So it is difficult for single feature to 


/ \,surround inhibition 

( fa X re?i ° n(nCRF) 


A i*^i A 

IV ' ■ 1 I - f f 1 
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Fig. 2. The distance weighting function of surround inhibition. 


evaluate the homogeneity between center and surround, and is 
hard to generate precise suppression in night vision images. 

Experiments have shown that the inhibitory degree depends on 
the difference between optimal response orientation (preferred 
orientation) and stimulus orientation in neurons. Modulatory ef¬ 
fect is strongest when CRF and nCRF have similar preferred orien¬ 
tation [23-25] and it weakens with the increasing of the 
orientation difference [23,24,26,27]. Gray spatial frequency [25] 
and contrast [28] play similar roles. Thus the inhibition is strongest 
when accordant features in CRF and nCRF, it decreases or disap¬ 
pears when there are differences in center and surround. So we 
introduce multidimensional features analysis to present a multi¬ 
feature contrast (MFC) weighted inhibition model. 

Compared with high-quality image, signal noise ratio is low in 
LLL image and boundaries are blurry in infrared image. Contour 
influenced by serious noise in LLL image gives no obvious orienta¬ 
tion. But local spatial frequency around contour is more prominent 
than surroundings in global noise. Blurry contour in infrared image 
also has weak orientation response. But local contrast of contour is 
higher than surroundings. So we design a features selection unit, 
including spatial frequency SF, contrast CON, co-occurrence statis¬ 
tics of homogeneity cooH and correlation cooCOR (Appendix A is 
the computing of these features), to detect comprehensive differ¬ 
ences between nCRF and CRF. The feature vector of CRF is defined 
as x = [SF C , CON c , cooH c , cooCOR c ] T and feature vector of nCRF is de¬ 
fined as y = [SF nc , CON nc , cooH nc , cooCOR nc ] T . Here, SF C , SF nc , CON c , 
CON nc , cooH c , cooH nc , cooCOR c and cooCOR nc are normalized spatial 
frequency, contrast, co-occurrence statistics of homogeneity and 
correlation in regions of CRF S c and nCRF S nc respectively. MFC va¬ 
lue changes with the angle 0 A and Euclidean distance d A , which 
represent the direction and distance between feature vectors in 
feature space. 0 A and d A measure the contrast of multi-feature in 
center and surround. The MFC weighting is defined as follow. 


C0mfc(x,y) = exp 


(1 -COSfl A ) 2 +8i4 

2a\ 


( 8 ) 


cos 9a = MF 1 M ’ dA = l|x “ y|1 (9) 

Here si controls the proportion of d A and 0 A in multi-feature con¬ 
trast. The standard deviation of Gaussian distribution a A deter¬ 
mines the attenuation degree for MFC weighting co m / c (x,y) with 
multi-feature contrast. 

In Fig. 4(cl, c2), we compare the differences of center and sur¬ 
round windows in small and large scales from the aspects of orien¬ 
tation, spatial frequency, contrast, co-occurrence statistics of 
correlation and homogeneity. The sizes of center and surround 
windows are 3 x 3 and 7 x 7 in small scale, 7x7 and 21 x 21 in 
large scale. We can see that it is difficult to distinguish the center 
and surround in weak contour regions by only orientation, the dif¬ 
ference of orientation is small especially in large scale, but the 
other four features can discriminate better either in small scale 
or large scale. 

The multi-feature can distinguish center and surround well. So 
the MFC weighting defined in Eq. (8) constructed by the distance 
and angle of multi-feature vectors can measure the difference be¬ 
tween CRF and nCRF region effectively. In Fig. 4, it is evident that 
MFC value of contour point is outstanding (dl, d2) in situations 
of strong noise (bl) and low contrast (b2). Through analyzing the 
multi-feature in center and surround, pixels with similar feature 
vectors in CRF and nCRF have small 0 A and d A to gain high co m / c 
in homogeneous region. Whereas, contour pixels with different 
feature vectors get larger 0 A and d A to generate relatively lower 
cOrnfci which allows the MFC weighting of contour to stand out from 
background. 
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Multi-scale Iterative Attention Method: 

<x, =<r 0 +rjt 

^ R a (x,y,O t ) = -R a (x,y,^ >-*- ja/?^ (x,y,0, )-b t a (x,y,0,)r+cf e {x,y.0 t )| 
R„ (x,y)=max{/?„ )|i=0,l 


f 



Fig. 3. General flowchart of the proposed method. 



Fig. 4. Multi-feature contrast analysis, (al) and (a2) original image, (bl) and (b2) enlarged view of red box area with 21 x 21 pixels, (cl) and (c2) center and surround 
differences by different features in small and large scales, (dl) and (d2) grid graph of value 1 - co m f C in red box region. Parameters are set as s^ = 0.4, a A = 0.5, o = 3 for LLL 
image and o = 2.5 for infrared image. 


We also consider that inhibitory strength decreases 
with the increasing distance from nCRF pixels to CRF 
center m<j(x,y) and the orientation-selective mechanism 
cos(0,- - 6j), the output of MFC inhibition t a (x,y;6i) is fi¬ 
nally improved as follow. 

N e 

ta(x,y ; 0j) = C 0 mfc {x,y) ■ Y) cos(0j - Oj) ■ [a>a(x,y) * E a (x,y; 9j)] (10) 
i=i 

Here, 6 t and Qj are the orientations of pixel in CRF and nCRF. 


Compared with original inhibition co^x.y) * E a (x,y; 6i) in Eq. 
(7), the Eq. (10) co m/c (x,y) • ^ cos(0/ - Oj) ■ [co a (x,y) * £ ff (x,y ; Oj)] 

adds the contrast of orientation Ylj’l-i cos(0/ - 9j) and MFC weight¬ 
ing co m fc{x,y) to gain the ability of improving surround inhibition. 
In homogeneous area the MFC weighting is large, the inhibition 
is strong. In inhomogeneous area the MFC weighting is much smal¬ 
ler, the inhibition is weak. So can we suppress most of the texture 
regions and keep the main contours. 

Compared with orientation-selective mechanism in previous 
methods, MFC inhibition model has a higher robustness to apply 
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in night vision images by considering features of preferred orienta¬ 
tion, spatial frequency, contrast and co-occurrence statistics. As is 
shown in Fig. 5, the MFC inhibition model can suppress noise 
and texture more effectively to segregate contour from background 
with less false boundaries. Furthermore, introducing multi-feature 
analysis can reduce mutual inhibition among contour elements to 
strengthen contour response. 

2.4. FC facilitation model 

In Fig. 5 there are still false boundaries and cracked contours in 
LLL and infrared images due to imaging quality and excessive sur¬ 
rounding suppression. How to balance the suppression of back¬ 
ground and retention of contour is an essential problem. It is 
remarkable that contour grouping is dedicated to extract coherent 
structures in a field of randomly oriented elements. Thus, the prin¬ 
ciple of grouping can be used to strengthen contour response and 
organize local boundaries into integrated contour. 

In Gestalt rule [7], visual system tends to group consistent im¬ 
age features to higher-level structures through some rules, includ¬ 
ing that curvature constancy is prone to maintain the original 
curvature in curve extending; low curvature tends to connect seg¬ 
ments by curve with low curvature; local proximity is in favor of 
connecting segments with short distance. 

Geisler et al. [12,15] provided a maximum likelihood local 
grouping function of co-circularity rule for contour grouping in 
natural scenes. The co-circularity rule requires that connective ele¬ 
ments should be tangent to a circle. Thus optimal orientation of 
pixel A 6i (0 < 6i < n) and direction of pixel B 0/(0 < Oj < n) should 
follow Eq. (11) in Fig. 6. Here 3 is the deviation, which can be set as 
nlN e . 

Oj = 2 Oab — Of d= 3 0 < 2 Oab — Oj < 71 

e j = 26 AB -6 i + n±3 2e AB -6i<0 (11) 

Oj = 2 Oab -0 t -7l± 3 20 AB - 0 t ^ 71 



However, co-circularity constraint is not comprehensive 
according to Gestalt rule. We propose a fuzzy connection (FC) facil¬ 
itation model for contour integration. An optimized grouping con¬ 
straint, combining the co-circularity rule with the preference of 
curvature constancy, low curvature and local proximity in visual 
perception, is designed to connect local segments in consistent 
arrangement. Based on the optimized grouping constraint, a con¬ 
nection-probability from pixels in nCRF to center pixel of CRF is 
calculated to decide the contour grouping significance of center 
pixel. 

2.4.1. Connection-probability 

According to rules of low curvature and local proximity, the 
connection-probability function should attenuate with the increas¬ 
ing of distance and curvature. So we define connection-probability 
between pixel A and B in Fig. 6 as follow. 

P e ^(Mi) = exp ( 12 ) 



Fig. 5. Inhibition results by different methods. Parameters are set as si = 0.4, <r A = 0.5, o = 3 for LLL image (close shot LLL1 in first row) and o = 2.5 for infrared image (distance 
infraredl in second row and close shot infraredl in third row). 
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Fig. 7. Geometrical analysis of arc length and Euclidean distance connection. 

Here P B ^ A ( 0 it 9j ) is the connection vector, P B ^ A {9 it 9j ) is the va¬ 
lue of connection-probability, connection orientation is 
9 AB (0 ^ 0 AB < n), £ 2 >0 is the coefficient to adjust the proportion 
of distance and curvature in connection-probability, a affects the 
size of connection field. To keep the criterion of curvature con¬ 
stancy, co-circularity rule is still considered. So the geometrical 
relationship between pixel A and B should fit Eq. (11). Then the 


controls the proportion of curvature and arc length (0 < s 2 ^ 1), 
the larger the s 2 is, the flat the excitatory connection is. The devia¬ 
tion angle \9 t - 0 AB \ between e BA and e\ (in Fig. 6) impacts the prob¬ 
ability of excitatory connection, the larger the |0* - 9 AB \ is, the low 
the excitatory connection probability is. By analyzing the relation 
from g 2 and \9 t - 0 AB \ to connection field, we can see that the con¬ 
nection is significant only in a range of deviation angle. 

As a conclusion drawn in 12] that if the deviation angle \6i - 
0 AB | is more than 7i/4, it cannot guarantee that the curvature de¬ 
creases with the orientation contrast. So it is constrained that 
\9i - 0 AB | ^ 7r/4. However, according to the numerical analysis, we 
convincingly explain that the constraint can be extended to 
(| 9i- 9 ab | < 7i/4) u (371/4 ^ 1 9i - 9 AB \ ^ n) in FC facilitation model. 

In Fig. 8(a), when n/2 < \9 t - 9^ ^ n and |0,- - 9 AB \ ^ tt/ 2, the 
connection field is full symmetry. When \9 t - 9 AB \ ^ n/2, the smal¬ 
ler the orientation contrast between center and surround is, the 
smaller the curvature of co-circularity points is. When 
7i/2 < 1 9i - 9 ab | ^ 7i, the smaller the orientation contrast is, the lar¬ 
ger the curvature is. In Fig. 8(b), when we restrain £ 2 = 0.1 and 
P B ^ A ^ 0.3 the maximum deviation angle is 43.6°. Regardless of 
the value of e 2 there is barely reaction in the range of n/4 < |0 Z - - 
9 ab | < 37i/4, so the new constraint 

(| 9i - 9 ab | ^ 71/4) u (371/4 ^ 1 9i - 9 ab \ ^ n) can both ensure the 
effective range of connection and reduce computing work. We as¬ 
sume that connection happens only when 0 ^ |0 f - 9 AB \ ^ n/4 and 
371/4 ^ 1 9i - 9 ab \ ^ 7i, Eq. (11) is modified as follow. 


0j = 29 ab -9i±S,(p = ( 0t - 0j)/2 0 ^9i< 71/2, 0^\9 i -9 AB \^ n/4 

or 7i/2 ^ 9i < 71,371/4 <| 9i - 9 ab |< n 
9j = 29 ab -9i + n±d,(p = n/2 + (0* - 0>)/2 0 <| 0 Z - 9 ab |< n/4 
9 j = 29 ab - Oi - n±S,cp = n/2 - (0,- - 9j)/2 3n/4 0* - 9 AB |< n 


(13) 


spatial distance between pixel A and B is d = s/m 2 + n 2 = 2 rsincp, 
arc length is s = cpd/ sin (p, and curvature is k = 1/r = 2 sin cp/d. 

Compared with [12], we use arc length instead of Euclidean dis¬ 
tance to measure local proximity. Pixel in discontinuous contour is 
actually connected by curve along the tangent direction, so the 
connection distance between two points represented by arc length 
is more accurate than Euclidean distance. 

As is shown in Fig. 7 (d < n/4), if we use the Euclidean distance 
to evaluate the local proximity, point 0 tends to connect with point 
Q for d op > d oq . If we use the arc length to evaluate the local prox¬ 
imity, O tends to connect with P for s op < s oq . According to the char¬ 
acteristic of low-curvature, O is more suitable to connect with P. 
Therefore the arc length connection is more accurate and accor¬ 
dant with Gestalt rules. Actually, in Fig. 7, there are s = dr, d op > d oq , 
So, = § a, s„ p = &(a + Jj) = Oa f"**? 5 * ), so s op > s„, when d > n/4 
and s op < s oq when d < n/4. 


Besides, compare the connection field of our facilitation model 
with model in [12] (Fig. 8(c)), connection in [12] tends to be more 
flat, whose connection field is similar to our model with e 2 = 1. 
Then gaps may occur at corners with high-curvature. £ 2 is adjust¬ 
able in FC facilitation model, so both smooth or high-curvature 
connection can be achieved. The value of e 2 will be discussed be¬ 
low. As is shown in Fig. 9, it is better connected at inflexion points 
by our model. 

Furthermore, if the surround deviates from the orientation axis 
of the center excessively, it will fail to form a co-aligned configura¬ 
tion. So we introduce a new constraint into the connection-proba¬ 
bility, which is the projection from connection direction f BA to 
preferred direction e\ at A. Then the connection-probability is 
amended as P B ^ A (di,Oj) = exp(-^i^) • cos (|0 f - 0 AB r), \0 t - 
9 ab | + = min(|0/ - 0^1, n - |0* - 0^ B |) The connection-probability is fi¬ 
nally calculated as: 


P B ^ A (9i, 9j) 


ex p (-^#) • COS(|d f - d A B| + ) (0 < |0i - d AB | < 71/4) U (377/4 < 1 0i - 0 AB | < 71) 

eX P ( _ i&) (l 0i - M = °) U (l 0 i - 0 Ab| = 77) 


(14) 


2.4.2. Constraint condition 

Fig. 8 is the distribution field of the excitatory connection prob¬ 
ability for short line (unit length), which is computed by Eq. (12). In 
Fig. 8, connection-probability attenuates with the length of smooth 
path increasing and tends to keep the direction of straight line. £ 2 


2.4.3. Grouping significance 

All points (e.g. pixel B in Fig. 6) in the connection field of pixel A 
will connect to A. The contour grouping significance of pixel A is 
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Fig. 8. The distribution field of the excitatory connection probability for short line, (a = 0.5 0 < s 2 ^ 1). 
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(a) (b) (c) 


Fig. 9. Comparison of different facilitation model for contour extraction, (a) is an infrared image, (b) is facilitation model in [12], (c) is FC facilitation model in this paper 
(£2 = 0.6). 


got by superposing all connection vectors at A. But only when the 
nCRF points meet the geometrical relationship in Eq. (13) with the 
center pixel in CRF, can they have probability to connect with cen¬ 
ter pixel. 

In addition, the nCRF points should have prominent Gabor en¬ 
ergy in their orientation 9j to avoid mutual facilitation in homoge¬ 
neous region. So the contour grouping significance oj fc {x, y, Oi) of A 
is defined as the combination of optimized grouping constraint and 
local orientation energy of B. 

co fc (x,y,8 i ) = ]T P B ^A(di, dj) ■ E a (B- dj) (15) 

BeSncje[0,N g ) 

The existing facilitation models only consider the superposed 
connection from surrounding pixels to center pixel. That is only 
the situation for center pixel to be connected is calculated, without 
considering the attribute of center pixel itself. For region of back¬ 
ground texture, the Gabor energy of each surrounding pixel in each 
direction is weak. For edge area of the target, the Gabor energy of 
surrounding pixel is strong only in part of directions. However, the 
facilitation effect is a superposed result, which is computed by 
accumulate the connection-probability from each surrounding pix¬ 
el in each direction toward the center pixel. This could cause sim¬ 
ilar facilitation response in contour area and homogenous region. 

In this paper we consider that if the central pixel is potential 
contour the facilitation response increases. On basis of the super¬ 
posed connection of surrounding pixels, we consider the Gabor en¬ 


ergy of center pixel. So the FC facilitation result depends on 
contour grouping significance and Gabor energy of center pixel. 
Accordingly, the function of FC facilitation f a (x,y; Of) can be calcu¬ 
lated as follow. 

/<r(x,y; 9i) = co f c(x,y, 0,) • E a (x,y ; 0,-) (16) 

Center pixel itself with strong Gabor energy (potential contour) 
has a higher probability to be connected. Central pixel itself with 
weak Gabor energy (background texture) has a low probability to 
be connected. This can further distinguish between facilitation re¬ 
sponse of background and contour. As is shown in Fig. 10, com¬ 
pared with the blurry facilitation boundaries by [12], our model 
has more sharpened edges and the facilitation contour is more 
remarkable. 

In night vision images, as for the pixels in internal texture re¬ 
gion, they and their connection points are in random direction 
and have little Gabor energy because of the homogeneity, so they 
get weak facilitation response. The pixels in boundary connect 
mutually with large Gabor energy and consistent distribution to 
achieve high grouping significance and strong facilitation response. 
In Fig. 10, it is effective for FC facilitation to keep regular grouped 
contours and reduce noise interference. In addition, fuzzy connec¬ 
tion will contribute to enhance the response of weak contours and 
fill in the gap between boundaries. Thus the FC facilitation model 
performs well in contour extraction and visuospatial integration. 
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spatial facilitation in [12] 


c-- r> ( 


FC facilitation 

Fig. 10. Facilitation results in night vision images. Parameters are set as & 2 = 0.6, o = 3 for LLL image and o = 2.5 for infrared image. 


2.5. Multi-scale iterative attention method 

Physiology studies indicate that neuron in visual cortex accom¬ 
plishes a particular visual task in a dynamic and flexible way 29]. 
When the peripheral region of CRF is long time stimulated, the CRF 
area will be multiple times larger [30,31]. Inspired by this biolog¬ 
ical characteristic, we propose a multi-scale iterative attention 
method to achieve dynamic modulation of nCRF. 

The area of CRF depends on scale factor a. Large-scale allows 
larger scope of interaction to extract contour of big target. But it 
may weaken contour of small target. Small-scale makes the modu¬ 
lation process more localized, which is conducive to keep small 
target. But it also retains some texture and noise. The proposed 
method increases the scale during iteration process to notice sali¬ 
ent contour of targets in different sizes and inhibit background. 

We define variable a t to replace o in the proposed models, 
which increases linearly with the iteration number: a t = cr 0 + rjt. 
In each step of iteration process, modulation result in each orienta¬ 
tion is given as R at (x,y,0i). The final output is the maximum re¬ 
sponse over all orientations. 

J t Ra t (X,y,0i) = -Ra,(X,y,0i) 

+ \aR„ t (x,y, Oi) -bt a ,(x,y, 0() + cf at {x,y, 0i)| + (17) 

Rcrjx.y) = max{R r!t (x,y-,e i )\i = 0A,....N„- 1} (18) 

Here a, b and c control the proportion of inhibition and facilita¬ 
tion, the initial value of R at is set as E ao . 

In this multi-scale iterative attention method, the previous out¬ 
put at smaller scale is the input of next larger scale. It is difficult to 
control single scale adaptively to adapt for all targets and obtain 
the accurate and complete contour of target with any size. For 
small scale, it can inhibit homogeneous regions and retain contours 
of small target with fewer gaps. The facilitation is prone to obtain 
more natural and continuous gaps connection, but it remains a 
great deal of texture boundaries. For large scale, it has the ability 
to suppress texture to a greater degree and retain main contours, 
but it tends to lose small object contour and bring in big gaps, 
which is hard for facilitation to connect naturally and 
continuously. 

Therefore, we operate the previous output at smaller scale with 
a larger scale and increase the scale in iteration. The target 


contours are continuous and texture boundaries are scattered. So 
that small-scale inhibition makes the contour of small target be 
retained, small-scale facilitation makes the contour intact; 
large-scale keeps contour of large target and inhibit texture in a 
large extent. Meanwhile the continuous small target contours 
and scattered texture boundaries in surrounding are obvious 
different, which preserves contour of small target in large-scale 
inhibition. Combined with facilitation, the response of small target 
contour is growing through iteration process. Thus the iteration 
process has the ability to constantly suppress texture boundaries, 
enhance contour of small target also keep contour of large target. 
Increasing facilitation scale gradually will make discontinuous 
contours connection more natural and accurate. 

As is shown in Fig. 11, in small-scale contour of small target is 
kept ((a) and (b) green circle) with some residual texture, large- 
scale loses contour of small target ((c) and (d) green circle) but has 
less residual texture, which is an irreconcilable problem for contour 
extraction of targets in different sizes by single scale. However, small 
target is strengthened and large target is highlight in multi-scale 
iterative attention method ((e) green circle). 

A Gaussian noise added circle is used to illustrate the effect of 
iteration in contour extraction, as is shown in Fig. 12. During the 
process of multi-scale analysis, MFC inhibition weakens homoge¬ 
neous surrounding and irregular noises gradually to stand out 
main contour, FC facilitation strengthens excitatory connection to 
ensure the continuity of contour. The method performs well in 
extracting contour from serious noises. 

3. Models analysis 

3.2. For the facilitation model, why is the term “fuzzy connection” 
used? How was the “fuzzy" strategy employed during grouping 
contour segments? 

First, the inhibition model may lead to contour disconnection. 
Contour discontinuity in serious condition must be connected. 
Second, if we analyze the grouping of discontinuous contour after 
suppression to determine the connection, the connection informa¬ 
tion is completely depends on the suppression outputs, leading to 


1 For interpretation of color in Figs. 11 and 13, the reader is referred to the web 
version of this article. 
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(a) (b) (c) (d) (e) 


Fig. 11. Comparison of the small-scale model, large-scale model and the multi-scale iterative attention method, (a) and (b) are inhibition and inhibition + facilitation results 
in small scale, (c) and (d) are inhibition and inhibition + facilitation results in large scale, (e) is the iterative result. 



t=8 t=10 


Fig. 12. Iterative attention results of the analog data. Parameters are set as a 0 = 0.5, r\ = 0.35, a = 1, b = 0.6, c = 0.4, t = 10. 


possible superposed errors. If we analyze the information of group¬ 
ing facilitation directly on original image and calculate the proba¬ 
bility of all potential contour pixels to be connected (It can be seen 
from the FC facilitation model that the potential contour segments 
have high probability to be connected), the facilitation effect is not 
dependent on the inhibition outputs. It will reduce superposed 
errors, which can improve the accuracy of contour extraction. Also 
inhibition and facilitation effects are independent, parallel process¬ 
ing can be achieved to improve the computational efficiency. 

Fuzzy connection can realize the analysis of grouping facilita¬ 
tion on original image. From the formulas in FC facilitation model, 
we can see that any point in image can receive the “vote” from its 
neighborhood. Accumulation of the “vote” is the grouping connec¬ 
tion probability at this point. So each point in image has not the 
only state to be connected or not, but each one is connecting or 
connected to a certain probability, which is the concept of 
“fuzzy connection”. The connection-probability is calculated for 
each point as “fuzzy” measure to assess its grouping significance, 
and is combined with inhibition results to enhance contour 
discrimination. 

3.2. Experiment-based parameters analysis and discussion 

In this paper, for MFC inhibition the proportionality coefficient 
and standard deviation of Gaussian distribution are set equal to 
£i = 0.4 and a A = 0.5. In FC facilitation the weighting of curvature 


is set equal to e 2 = 0.6. In multi-scale iterative attention the initial 
scale factor, growth rate of scale, iteration number and proportion 
of inhibition and facilitation are set as a 0 = 0.5, rj = 0.35, t= 10 and 
a = 1, b = 0.6, c = 0.4, respectively. It seems to involve many free 
parameters, but only three critical parameters affect the models’ 
performance. 

In contours output, non-maximum suppression and binaryza- 
tion is performed to remove false boundaries and select the salient 
contours with strong modulation response and long length. To dis¬ 
cuss the influence of parameters, we use the definitions proposed 
by Grigorescu et al. [2]. c t is the pixel set of truth salient contours; 
b t is the pixel set of truth background edges; c e is the pixel set of ex¬ 
tracted contours; b e is the pixel set of extracted background edges. 
Then correctly detected pixel set of contour is c e n c t ; false negatives 
are given by b e n c t ; false positives are given by c e n b t . We define 
three new measures in this paper: accurate rate A r = num{c e n c t )/ 
num(c t + b t ), false rate F r = num{b e n c t )/rnim(c t + b t ) + num(c e n b t )l 
num(c e n c t ), capacity of contour extraction C = A r -F r . 

(1) The weighting of curvature s 2 decides the contour structure 
in FC facilitation model. The greater the curvature is, the 
smoother the contour is. Combined with Fig. 8 we discuss 
how the accurate rate A r changes with s 2 in Fig. 13(a). No 
matter what the value of s 2 is, the straight line connection 
has the maximum probability to be executed. There are 
mainly straight line contours in distance infraredl image 






174 


J. Han et al.)Infrared Physics 6f Technology 63 (2014) 165-177 



weighting of curvature 


(h) 20 l 



—*— close shot LLL1 
distance infrared 1 
—♦— close shot infrared 1 


■ computing time 



i— 1 —i— 1 —i— 1 —i— 1 —i— 1 —i— 1 —i—'—i— 1 —i— 1 — r~ 

2 4 6 8 10 12 14 16 18 20 

iterations 


Fig. 13. Analyze the impact of parameters on models’ performance, (a) The curve for accurate rate A r changing with weighting of curvature e 2 . (b) The curve for capacity of 
contour extraction C changing with iteration number t . 



(a) (b) (c) (d) (e) (f) 


Fig. 14. The outputs of contour extraction in each step during iteration, (a) and (b) is the inhibition and inhibition + facilitation results in first iteration step, (c) and (d) is the 
inhibition and inhibition + facilitation results in sixth iteration step, (e) is the iterative result, (f) is the binaryzation output of contours. Parameters are set as 
< 7 0 = 0.5, rj = 0.35, a = 1, b = 0.6, c = 0.4, t= 10. 


(image in second row of Fig. 5), so the larger the s 2 is, the 
higher facilitation accuracy is (green line). There are many 
high-curvature connection points in close shot infraredl 
image (image in third row of Fig. 5), making it achieves bet¬ 
ter facilitation result when s 2 is small (blue line). There are 
mainly curve contour in close shot LLL1 (image in first row 
of Fig. 5), g 2 ranges between the above two (red line). For 
the unknown night vision image, it is difficult to predict 
the contour structure, so we take a compromise in experi¬ 
ments and set the weighting as s 2 = 0.6. 

(2) The iteration number t decides the accurate and false rate of 
contour extraction, also it determines the computing time of 
the multi-scale iterative attention method. We discuss how 
the capacity of contour extraction C changes with t; in 
Fig. 13(b). When t is low, the modulatory coverage deter¬ 
mined by scale factor is small and both contours and some 
textures are all retained, making A r is high, but F r is much 
greater, so A r - F r is low. When t increases the scale is rising, 
response of part contours are weakened, meanwhile texture 


inhibition is enhanced, leading to A r and F r both reduce. The 
combinatorial role of inhibition and facilitation makes the 
degree of texture inhibition large, but the degree of weaken¬ 
ing in contour response is small. Thus the reducing of F r is 
much rapider than A r to gain a maximum value for A r - F r 
in best iteration output. In subsequent iteration t further 
increases, the scale will be larger than many targets, so F r 
is further reduced and A r is also falling down. In this situa¬ 
tion, residual textures are little so F r reduces slowly. Despite 
the facilitation contours response are incessantly weakened 
due to mutual inhibition between each other, thus A r 
decreases seriously. So A r - F r is dropped. Besides, the larger 
t is, the greater the computing time is (orange line). As is 
shown in Fig. 13(b) (red, green and blue lines), when t= 10 
the contour extraction obtains best performance in different 
image scenes. 

(3) The proportion of inhibition b and facilitation c also deter¬ 
mine the performance of contour extraction. The response 
of targets may be weak when excessive inhibition in 
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original image 


butterfly-shaped 
inhibition 


two-scale inhibition 


modulation model 
in [12] 


presented method 


Fig. 15. Performance of contour extraction from night vision images in natural scenes by different methods. Rows from top to bottom are images of noisy hyena, close shot 
infrared2, distance infrared2, distance LLL1 and distance LLL2 respectively. Parameters are set as s-i = 0.4, cr A = 0.5, s 2 = 0.6, a 0 = 0.5, rj = 0.35, a = 1, b = 0.6, c = 0.4, t = 10. 


iteration process. Angular points may be ignored for the 
preference of smooth connection when facilitation plays a 
dominating role. So the value of b and c depends on the con¬ 
tour structure of significant objects in scene. Buildings 
scenes (many straight line contours) can set a larger propor¬ 
tion of inhibition, scenes with people and vehicles (complex 
shape of target contour) use a larger proportion of facilita¬ 
tion. We will further research on selective and adaptive 
modulation between inhibition and facilitation based on 
scene. 

4. Empirical evaluation 

Our method is applied to extract salient contour from night 
vision images. As is shown in Fig. 14, the output of small-scale inhi¬ 
bition has many boundaries of texture (Fig. 14(a)). The grouping 
significance of clutter texture is lower and grouping significance 


of well-organized contour is much higher after inhibition + 
facilitation effect (Fig. 14(b)). Gradually increasing the iteration 
number to grow the scale, background in the output of small-scale 
is further weakened by inhibition in large-scale (Fig. 14(c)). The 
inhibition + facilitation role further enhances the response of sali¬ 
ent contour (Fig. 14(d)). Finally Fig. 14(e) is the iteration result, 
Fig. 14(f) is the binaryzation output of contours. The results dem¬ 
onstrate the robustness of multi-scale analysis. 

To further illustrate the property of the presented method, we 
compare the effect of our method with the butterfly-shaped inhibi¬ 
tion, two-scale inhibition and modulation model in 12] on Gauss¬ 
ian noise added visible image, real LLL and infrared images in 
different natural scenes. As is shown in Fig. 15, despite of the dif¬ 
ferent texture structure, noise distribution, scene complexity and 
target significance, the presented method can inhibit surrounding, 
extract contours of different targets with multi-size and maintain 
the integrity of contour better than the other models. So in nature 
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Fig. 16. Statistical box-and-whisker plots of the performance. B denotes the butterfly-shaped inhibition, S the two-scale inhibition, M the modulation model in [12], P the 
presented method. 
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scenes, our method performs well in separating edges of targets 
from heavy noises and abundant background and connecting dis¬ 
continuous contour segments in LLL and infrared images to 
emphasize salient contours. 

To test the robustness of the proposed model, we conducted a 
statistical analysis on the performance of 180 groups of different 
parameter combinations mentioned above (e 2 = [0:0.2:1 ], 
t= [2:2:20], b/c = [0.5,1,2]) to calculate box-and-whisker plots 
for the butterfly-shaped inhibition, two-scale inhibition, modula¬ 
tion model in [12 and presented model. 

The performance of contour extraction defined in [2] is 
P = num{c e n c t )/(num(c e n c t ) + num(b e n c t ) + num(c e n b t )). It is 
used to quantitatively evaluate the contour extraction. Fig. 16 
shows this statistical box-and-whisker plot for ten images men¬ 
tioned above in this study. In each plot, the top end of a whisker 
represents the best performance among the 180 different results 
for each image, and the horizontal red line in the box of a whisker 
shows the median value of 180 performance measures. The top and 
bottom edges of each box reflect the upper and lower median, 
respectively. 

For all test images, the top end of whisker of the presented 
model is obviously higher than that of other models. More impor¬ 
tantly, the presented model produces a significantly higher median 
performance measure, and the box height (i.e., the range between 
the upper and lower medians) of the presented model is smaller 
than that of most boxes of other models, which is particularly obvi¬ 
ous in real LLL and infrared images. It reveals that our model per¬ 
forms better and has higher robustness in extracting salient 
contour from cluttered scene in night vision images. 


5. Conclusions 

In this work, according to characteristics of night vision images, 
we improve the receptive field models based on perception mech¬ 
anism and propose an effective contour extraction method to sup¬ 
press noise and texture and detect salient contour from LLL and 
infrared images in natural scene. 


The optimized modulation method based on compound recep¬ 
tive field coordinates two aspects of surround inhibition and 
grouping facilitation. As for surround inhibition, we first bring for¬ 
ward a multi-feature contrast weighted inhibition model. Different 
from the orientation-selective mechanism, we focus on features in 
LLL and infrared images to evaluate the correlation of center and 
surround in feature space and weighting inhibition by the multi¬ 
feature difference. The MFC weighted inhibition model can 
eliminate details effectively and lower the possibility of mutual 
inhibition among contour elements. As for grouping facilitation, 
we innovatively present a fuzzy connection facilitation Model. 
Other from co-circularity method, we establish grouping facilita¬ 
tion with more comprehensive perception constraints to calculate 
connection-probability between center and surround and estimate 
grouping significance for each pixel. The FC facilitation model is 
consistent with Gestalt rule to achieve precise contour connection. 
Finally, we design a visual inspired multi-scale iterative attention 
method to accomplish hierarchical attention from details to main 
contours by changing scale factor in iteration process dynamically. 

However, the response of targets may be weak due to the 
inhibition mechanism in iteration process. Angular points may be 
ignored for the preference of smooth connection when facilitation 
plays a dominating role. So we will research on selective and adap¬ 
tive modulation between inhibition and facilitation in future work, 
that is how to decide the value of a, b and c intelligently. 
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Appendix A. Computation of multiple features 

Consider a given local image of size MxN, where M is number 
of rows and N is number of columns. The spatial frequency SF is de¬ 
fined in [32]. The row and column frequencies are given by 

RowJreq = f JL [F(i,j) - F(i,j - 1 )] 2 and Column Jreq = 
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\J Mjv E“o’ ET=o [F(iJ) - F(i - 1 J)] 2 • The total frequency is then 

SF = yj (Row Freq) 2 + (Column Freq) 2 . This definition of frequency 
in the spatial domain indicates the overall activity level in an 
image. 

Contrast CON measures the local variations in center and sur¬ 
round windows, which is actually defined as the local standard 

deviation CON = ~ /tf. where fi is the local 

gray average. 

Co-occurrence statistics evaluate the joint probability occur¬ 
rence of the specified pixel pairs in gray-level co-occurrence matrix 
GLCM. In this paper, we consider GLCM in four directions 
(0,45,90,135) with offset of two pixels. 

Co-occurrence statistics of correlation cooCOR measures the 
similarity of elements in rows or columns in GLCM, which can re¬ 
flect the local gray correlation in image. cooCOR is calculated as 

cooCOR = ZtiZU where G is the GLCM matrix ' k is 

row and column number of G, u t = - C(i,j), 

uj = ZLZfj ■ C(iJ), sf = ZLZ jL,G(U)(i - u,) 2 , sf = Zl,ZU 
C(i.j)(j - Uj) 2 . cooCOR is big when the elements are uniform in 
GLCM and is small when the elements vary widely in GLCM. So it 
evaluates the consistency of image texture. 

Co-occurrence statistics of homogeneity cooH measures the 
closeness of the distribution of elements in GLCM to the diagonal. 
cooH is calculated as cooH = Sf=i i+n-jf 9 diagonal ele¬ 
ments in GLCM lead to big cooH , then image region with consistent 
gray has large cooH. So cooH evaluates the local homogeneity of 
image texture. 
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